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This report describes common and unique approaches that Regional Educational 
Laboratory (REL) Midwest Region states have adopted in developing and implementing 
their quality rating and improvement systems (QRISs). A QRIS is a method for assessing, 
improving, and communicating the quality of early childhood education and care 
providers. The study examined the ways that Race to the Top-Early Learning Challenge 
grants have shifted the QRIS landscape and how states have been influenced by the 
grant process in developing their QRISs, incorporating observations into their QRISs, 
using alternate pathways to ratings, and incentivizing providers to participate in QRISs. 
The report provides a knowledge-sharing outlet for states in the REL Midwest Region 
and beyond that may inform potential QRIS revisions. 


Why this study? 


Recent federal and state policies, such as the Race to the Top-Early Learning Challenge and the Pre- 
school for All initiative, 1 recognize the benefits of high-quality early childhood education programs and 
aim to improve the quality of such programs nationally. The Race to the Top-Early Learning Challenge 
grants (awarded in 2011, 2012, and 2013) supported states in their efforts to improve program quality — 
including a requirement for states to launch, revise, or expand a quality rating and improvement system 
(QRIS; box 1) that aligned with the grant’s QRIS guidelines. As a result, QRISs have expanded rapidly 
across the country, with new or revised QRIS initiatives in states that received Race to the Top-Early 
Learning Challenge awards as well as those that applied for funding but did not receive a grant. 
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Box 1. Key features of a quality rating and improvement system 


A quality rating and improvement system (QRIS) is a method to assess, improve, and communicate the quality of 
early childhood education and care providers (Mitchell, 2005). QRISs have the following key features: 

• Ratings typically range from 1 to 4 or from 1 to 5, with the highest number indicating the highest quality. 

• A QRIS usually includes measures of quality that incorporate self-ratings and direct observations of quality. 

• Most QRISs measure multiple aspects of early childhood education quality — most commonly staff qualifica- 
tions, curriculum, environment, administration and management, and health and safety. 

• Most QRISs are voluntary and are for licensed early childhood education providers that include center- and 
home-based care. 


Although 49 states implement a QRIS in some form (map 1), each QRIS differs in its approach to defin- 
ing, rating, supporting, and communicating quality. Because of this variability, QRIS administrators in REL 
Midwest Region states * 1 2 3 expressed an interest in sharing information about each state’s approach to implement- 
ing a QRIS as well as each state’s challenges and successes in order to inform the development of their own 
systems. For example, each state takes a different approach to incorporating direct observations of quality, using 
alternate pathways, and incentivizing providers (QRIS Compendium, 2015). QRIS administrators are interest- 
ed in other states’ strategies regarding direct observations because they are logistically challenging and costly 
to administer, but they may provide information about the program environment and adult-child interactions 
that may be difficult to capture through self-reported data or document reviews. QRIS administrators are also 
interested in how other states have used alternate pathways, which provide automatic or accelerated rating 
procedures for programs that meet quality standards developed for accreditation or public preschool funding. 
Alternate pathways can reduce the burden of QRIS participation for providers and administrators by reducing 
the need for additional data collection, but they may not represent quality in a way that is consistent with the 
full rating process. In addition, states with voluntary QRISs may need to offer incentives to encourage partici- 
pation, such as tiered reimbursement rates for child care subsidies or financial bonuses for rating increases, and 
QRIS administrators are interested in the strategies other states have used to incentivize participation. 

This report summarizes REL Midwest Region states’ key strategies used in developing and implementing 
their QRISs as well as the lessons they learned in the process. It is meant to be a resource for QRIS adminis- 
trators as states refine their QRISs. 

The study team conducted interviews with REL Midwest Region state officials, such as directors or manag- 
ers of state QRISs, to answer four questions about developing and implementing a QRIS: 

1. How did the Race to the Top-Early Learning Challenge process influence REL Midwest Region states’ 
development and implementation of QRISs? 

2. How did REL Midwest Region states incorporate direct observations of quality into their QRISs? 

3. How did REL Midwest Region states incorporate alternate pathways to ratings within their QRISs? 

4- How did REL Midwest Region states financially incentivize providers to participate in their QRISs, 
including the use of tiered subsidy reimbursement? 

The report also provides details on how REL Midwest Region states are revising their QRISs and additional 
resources to help states develop and refine their QRISs. Box 2 summarizes the data and methods used to 
conduct the study, and appendix A provides further detail. 
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Map 1. Status of quality rating and improvement systems by jurisdiction, February 2015 




Planning 

Requires legislative action to implement a 
quality rating and improvement system 


Note: Status is as of February 2015; a state's status may have since changed. 

a. Indicates that QRIS is not statewide and is instead administered by counties or other local or regional agencies. For example, in 
Florida 14 of the 30 local early learning coalitions operate voluntary QRIS programs. 

Source: Data provided by the QRIS National Learning Network and Build Initiative on the current status of QRIS and jurisdictions 
(http://www.buildinitiative.org/Thelssues/EarlyLearning/QualityQRIS.aspx). 


Box 2. Data and methods 

The study team first reviewed relevant documents such as state quality rating and improvement system (QRIS) 
websites, provider application templates, explanations of ratings structures, summary explanations of QRISs for 
families, and published research reports studying QRISs in the seven Regional Educational Laboratory Midwest 
Region states. Study team members reviewed the documents to familiarize themselves with the details of each 
state’s QRIS to ask appropriate follow-up or probing questions during interviews in the next stage of the study. 

The study team conducted a 90-minute semi-structured interview with representatives knowledgeable about 
the development and implementation of the QRIS in each state (see appendix B for the interview protocol). 
Seven interviews with eight respondents were conducted. Respondents included staff from the state agency 
that administers the QRIS, which varied by state. 

The study team qualitatively analyzed data collected during the interviews by coding the data using qual- 
itative analysis software to identify topics and themes. Study team members then examined both sources of 
data across all seven states to find similarities and differences in how they approach QRIS development and 
implementation. 
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What the study found 


All seven states in the REL Midwest Region have implemented a QRIS and — consistent with trends across 
the country — vary in their approaches to quantifying high-quality early childhood education. For example, 
states have a different number of rating levels and use different approaches for calculating ratings. Further- 
more, states began implementing their QRISs at different times. Information about each state’s QRIS can 
be found on the states’ websites (table 1). 

The process of applying for a Race to the Top-Early Learning Challenge grant helped advance development of a quality 
rating and improvement system, even in states that were not awarded funds, but capacity and funding to operate and 
sustain the systems remain challenges 

Interviewees reported that although the impetus for a QRIS varied by state, all REF Midwest Region states 
benefited from the Race to the Top-Early beaming Challenge grant process to build support for a statewide 
QRIS — even states that were not awarded a grant. One state QRIS administrator said, “I can’t empha- 
size enough how much of a catalyst the Early Learning Challenge grant was for all of this. [Otherwise] 
there would not have been political will to do this.” The grant application process motivated REL Midwest 


Table 1. Information on each Regional Educational Laboratory Midwest Region state’s quality 
rating and improvement system 


State 

Name of quality 
rating and 
improvement 
system 

Year implemented 

Number 
of rating 
levels 

Calculation 
approach 
for ratings 3 

Website 

Illinois 

ExceleRate 

2007 as Quality Counts, 

2013 as ExceleRate 

4 with 

Circles of 
Quality 11 

Hybrid 

http://www.excelerateillinois.com 

Indiana 

Paths to QUALITY 

2001 

4 

Building block 

http://childcareindiana.org 

Iowa 

Iowa Quality 

Rating System 

2006 

5 

Hybrid 

http://dhs.iowa.gov/iqrs 

Michigan 

Great Start to 

Quality 

2005 initial development, 
2009 pilot, 

2012 statewide 

5 

Point (initially), 
hybrid (as of 
2013) 

http://greatstarttoquality.org 

Minnesota 

Parent Aware 

2007 pilot, 

2012 county, 

2015 statewide 

4 

Hybrid 

http://parentaware.org 

Ohio 

Step Up to Quality 

2006 

5 

Hybrid 

http://www.earlychildhoodohio.org/ 

sutq.php 

Wisconsin 

YoungStar 

2010 

5 

Hybrid 

http://dcf.wisconsin.gov/youngstar 


a. The point approach assigns values to a variety of quality standards or domains (Zellman & Perlman, 2008), child care providers 
earn points for meeting specific quality indicators, and quality ratings are based on minimum required points earned for any stan- 
dard or domain (Tout et at, 2010). The building block approach uses sequential levels that can be reached only by meeting all the 
criteria within a level and the levels preceding it (Tout et al., 2010). The hybrid approach combines elements of both the building 
block and point approaches. 

b. Illinois uses four unique Circle of Quality designations or ratings. The first Circle of Quality designation, Licensed Circle of 
Quality, means that the provider meets state licensing standards for quality. The second, Bronze Circle of Quality, refers to the 
qualifications of provider staff; teachers and administrators at such providers have completed ExceleRate Illinois trainings and 
met staff qualifications and participate in continuous quality improvement. Providers that receive the third rating, the Silver Circle 
of Quality, either meet or exceed quality standards in learning environment and teaching quality, administrative standards, and 
staff training and education and are actively engaged in continuous quality improvement. Providers with the fourth rating, the Gold 
Circle of Quality, meet or exceed the highest quality standards in the three areas and are actively engaged in continuous quality 
improvement (ExceleRate Illinois, 2015). 

Source: Authors’ document review (see appendix A). 
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Region states to plan and develop a new QRIS, revise an existing QRIS, or expand the reach of their 
current QRIS. 

Minnesota and Ohio were awarded a Race to the Top-Early Learning Challenge grant in 2011, Illinois and 
Wisconsin were awarded a grant in 2012, and Michigan was awarded a grant in 2013 (table 2). Indiana and 
Iowa applied for a grant but were not awarded one. According to interviewees, both applying for and being 
awarded a grant shifted the landscape for QRIS implementation across states — for example: 

• Not being awarded a Race to the Top-Early Learning Challenge grant in 2011 prompted Illinois 
administrators to convene more stakeholders and experts to develop a stronger application for 2012. 
Illinois was awarded a grant in 2012, which allowed the state to deepen its QRIS development and 
revisions. Quality Counts, developed in 2007, grew into the new QRIS ExceleRate Illinois, which 
uses a unique Circle of Quality designation approach to rating early childhood education quality 
(see http://www.excelerateillinois.com). 

• Although Michigan was not awarded a Race to the Top-Early Learning Challenge grant until 
2013, its previous applications generated the political support needed for the Michigan Department 
of Education’s Office of Great Start to implement its comprehensive plan to expand the QRIS 
statewide in 2012. 

• The Race to the Top-Early Learning Challenge award provided Minnesota with the funding and 
political will to implement the existing QRIS. Previously, the QRIS had suffered from a lack of 
state funding, requiring a public-private partnership with the Minnesota Early Learning Founda- 
tion to support the pilot. 

• In Ohio the Race to the Top-Early Learning Challenge grant process brought together state 
agencies that oversee different birth'toage'5 programs but that had rarely worked together. Being 
awarded a grant generated the political will and funding needed to streamline the Step Up to 
Quality system and the Integrated Monitoring Process and Continuous Improvement Tool into a 
unified QRIS. 

Although the Race to the Top-Early Learning Challenge grants eased the challenges of limited state 
budgets and contributed to improved QRISs, interviewees from all states reported limited capacity to carry 
out the full scope of their QRISs. States struggled to launch and sustain systems, collect sufficient obser- 
vational data to compute ratings for all providers, and implement quality improvement supports (such as 


Table 2. Race to the Top-Early Learning Challenge applications and awards for the Regional 
Educational Laboratory Midwest Region states 


State 

Year or years of 
application 

Year grant 
was awarded 

Year grant 
funds expire 

Grant amount 

Amount allocated for quality 
rating and improvement 
system specific activities 

Illinois 

2011, 2012 

2012 (Phase 2) 

2017 

$52,498,043 

$46,163,852 

Indiana 

2013 

na 

na 

na 

na 

Iowa 

2011, 2012, 2013 

na 

na 

na 

na 

Michigan 

2011, 2012, 2013 

2013 (Phase 3) 

2018 

$51,737,456 

$29,331,566 

Minnesota 

2011 

2011 (Phase 1) 

2016 

$44,858,313 

$25,338,689 

Ohio 

2011, 2012 

2011 (Phase 1) 

2016 

$69,993,362 

$23,119,157 

Wisconsin 

2011, 2012 

2012 (Phase 2) 

2017 

$34,052,084 

$13,961,354 


na is not applicable because that state was not awarded a Race to the Top-Early Learning Challenge grant. 

Source: Authors' analysis of Race to the Top-Early Learning Challenge applications from all seven Regional Educational Laboratory 
Midwest Region states and U.S. Department of Education (2014). 
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technical assistance and staff trainings, scholarships to advance teacher credentials, and other related ini- 
tiatives). Michigan lawmakers stopped initial funding from supplemental appropriations after the state was 
awarded a Race to the Top-Early Learning Challenge grant, financially straining QRIS delivery in the 
state. 

Regional Educational Laboratory Midwest Region states have adopted a variety of approaches to strategically reduce 
the number of classrooms with direct observational assessments to balance the value of these data against its 
logistical and financial costs 

All seven REL Midwest Region states use direct observations of quality in combination with self-report 
measures to collect data on five broad dimensions of early childhood education quality: classroom environ- 
ment, staff qualifications, family and community engagement, instructional quality, and administration. 
Typically, the self-report measures focus on structural quality, such as teacher credentials and staff ratios, 
and the observations are used to measure process quality, such as instructional quality or teacher-child 
interactions. 

Three states in the REL Midwest Region use the Environment Rating Scales — Revised suite of assessments, 
which includes the Early Childhood Environment Rating Scales (Harms, Clifford, & Cryer, 1998) for preK 
classrooms and the Infant/Toddler Environment Rating Scales for classrooms with children ages 3 and 
younger. States use the Environment Rating Scales — Revised alone or in combination with the Classroom 
Assessment Scoring System (Pianta, La Paro, & Hamre, 2008). Michigan is the only state in the region 
that uses the Preschool Program Quality Assessment (High/Scope Educational Research Foundation, 2003; 
Jurkiewicz, 2003), and Indiana and Ohio each use their own state-developed observational assessment to 
measure process quality. 

The costs associated with conducting observations can be a challenge for states at all phases of QRIS 
implementation. In Illinois, Iowa, Michigan, Minnesota, and Wisconsin provider observations strained the 
state budget and were logistically difficult for state agencies to complete. States have started to identify 
personnel and financial resources to support independent observations and have adopted strategies to over- 
come funding and staffing limitations (table 3). For example, Iowa, Michigan, Minnesota, and Wisconsin 
have decided to conduct observations only for providers seeking the highest QRIS ratings. 

Another strategy is to limit the types of providers that receive ratings. Minnesota reserves the Classroom 
Assessment Scoring System for center-based preschool providers seeking a three- or four-star rating and 
does not conduct observations for family- or home-based care to help reduce observational costs. States 
also sample classrooms within providers to reduce the number of observations conducted at each provider, 
thereby decreasing the total cost of observations. Indiana observes at least a third of a provider’s class- 
rooms, whereas Ohio randomly selects half a provider’s classrooms to observe. To ensure that the rating is 
representative of the entire provider, both Indiana and Ohio observe at least one classroom per age group. 

The use of direct observational assessments, particularly the Classroom Assessment Scoring System, may 
also create challenges for QRIS participation among providers. Illinois and Indiana QRIS administrators 
voiced this concern about the system’s high thresholds, which providers in their states found difficult to 
meet. In Iowa there was concern that the observation requirement could discourage participation, since 
providers that were already accredited or participating in the Iowa Quality Preschool Program standards 
viewed the Environment Rating Scales — Revised observations to be duplicative. 

Despite the associated logistical and financial burdens, state agencies have expressed a commitment to 
continuing the use of standardized observational assessments. Interviewees stated that the widespread use 
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Table 3. Regional Educational Laboratory Midwest Region states’ observational measures and 
methods used to assess quality in their quality rating and improvement system 


State 

Observational measure 
of quality 

Providers that 
are eligible 

Approach to selecting 
classrooms for observations 

Illinois 

Classroom Assessment Scoring 
System, Environment Rating 

Scales — Revised 

Providers seeking a silver Circle 
of Quality conduct self-assessed 
observations; providers seeking 
a gold Circle of Quality must be 
validated by an outside assessor 

Classroom Assessment Scoring System: 
for Chicago Public School's Preschool 
for All program every classroom is rated, 
and for Head Start programs a third of 

classrooms at random 

Environment Rating Scales — Revised: a 
third of classrooms, with at least one per 
age group 

Indiana 

Indiana Classroom Observation 

Tool 

Providers seeking a higher rating 

A third of classrooms, with at least one per 
age group 

Iowa 

Environment Rating Scales — 

Revised 

Providers seeking the highest 
rating (level 5) 

A third of classrooms, with at least one per 
age group 

Michigan 

Preschool Program Quality 
Assessment (Form A) 

Providers seeking the highest 
two ratings (level 4 or 5) 

One classroom per age group 

Minnesota 

Classroom Assessment Scoring 
System 

Center-based preschool 
providers seeking the highest 
two ratings (three or four stars) 

A third of eligible classrooms 

Ohio 

Ohio Classroom Observation Tool 

Providers seeking the highest 
three ratings (three, four, or five 
stars) 

Half of classrooms, with at least one per 
age group 

Wisconsin 

Environment Rating Scales — 

Revised 

Providers seeking the highest 
rating (level 5) 

A third of classrooms for each age group 


Source: Authors' analysis based on a document review and transcripts from semi-structured interviews with quality rating and 
improvement system stakeholders (see appendix A). 


of direct observations has changed the definition of high-quality care by emphasizing the importance of 
process quality in early childhood education settings. One state QRIS administrator said, “By putting the 
[Classroom Assessment Scoring System] in there — by putting the focus on instructional support and curric- 
ulum and all that — we’re trying to push forward a real change in the field that’s very new. That is, the under- 
standing of . . . how much content in curriculum matters, how much intentional instruction matters.” States 
face the challenge of ensuring that their rating systems reflect a continuum of quality, given that the process 
quality expected at higher QRIS levels differs fundamentally from expected structural quality (box 3). 


Box 3. Process quality versus structural quality in early childhood education and quality rating and 
improvement systems 

Researchers have conceptualized early childhood education quality into two primary components: structural and 
process quality (see Vandell & Wolfe, 2000, for a review of the literature). 

• Structural quality includes the aspects of classrooms that can be regulated, such as class size, staff quali- 
fications, classroom materials, and teacher-child ratios. 

• Process quality includes more dynamic aspects of the classroom that involve the interactions between 
people in the classroom, typically including teacher-child or peer interactions. 

While most quality rating and improvement systems (QRISs) measure both structural and process quality, 
research suggests that process quality is linked most closely with child outcomes and that structural quality 
has a weaker or inconsistent relationship with child outcomes (Early et al., 2007; Sabol, Soliday Hong, Pianta, 
& Burchinal, 2013). Research also suggests that QRISs that combine both process and structural components 
into one overall rating had only weak relationships with student outcomes (Sabol et al., 2013). 
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Five states in the Regional Educational Laboratory Midwest Region use alternate pathways to rate certain early 
childhood education providers in their quality rating and improvement systems, most commonly accredited or state 
prekindergarten program providers 

Five REL Midwest Region states have sought flexibility in the requirements and processes by which pro- 
viders achieve a QRIS rating through alternate pathways, given that many early childhood education pro- 
viders already have quality standards and monitoring systems in place (for example, those for Head Start/ 
Early Head Start providers, state preK program providers, and accredited providers). Alternate pathways 
allow providers that already meet some form of quality standards (typically accreditation) to receive a QRIS 
rating without completing all the self-assessment or observation requirements. The assumption is that pro- 
viders that already have met the other standards or monitoring requirements are of high quality and there- 
fore can be accelerated to a higher QRIS rating. 

Alternate pathways are also used to encourage providers to participate in a QRIS because some providers 
report that completing the paperwork and observations is redundant with their own standards and a time 
burden. For example, providers that were already accredited or participating in the Iowa Quality Preschool 
Program standards viewed the Environment Rating Scales — Revised observations to be duplicative. Alter- 
nate pathways remove this burden and duplication and allow these select providers to enter the QRIS at an 
elevated rating (typically the second highest rating) due to the quality standards that they already meet in 
other early childhood education systems. 

Illinois, Michigan, Minnesota, Ohio, and Wisconsin all offer accelerated rating pathways for certain pro- 
viders to achieve higher QRIS ratings (table 4). The most common provider types to receive ratings through 
alternate pathways are those accredited by a national accreditation organization such as the National 
Association for the Education of Young Children. Three states also use alternate pathways to rate their 
state-funded preK providers (for example, Great Start Readiness providers in Michigan). 


Table 4. Five Regional Educational Laboratory Midwest Region states have specific eligibility 
conditions for accelerated ratings in their quality rating and improvement system 


Accreditation or program type 
eligible for accelerated ratings 

Illinois 

Michigan 

Minnesota 

Ohio 

Wisconsin 

American Montessori Society • 

Association of Montessori International-USA • 

Council on Accreditation 




• 

• 

National Accreditation Commission for Early Childhood Education 

• 


• 

• 

• 

National Afterschool Association • 

National Association for the Education of Young Children 

• 

• 

• 

• 

• 

National Association of Family Child Care • • 

National Early Childhood Program Accreditation 



• 

• 

• 

Head Start providers 


• 

• 



State-funded preK program providers 

(for example, Great Start Readiness providers in Michigan) 

• 

• 

• 



School-based providers • 


Note: Iowa and Indiana do not offer alternate pathways. 

Source: Authors' analysis based on a document review and transcripts from semi-structured interviews with quality rating and 
improvement system stakeholders (see appendix A). 
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Five states consider the reduced burden of collecting observational QRIS data in alternative pathways a 
cost-efficient way to rate providers as moderate or high quality because they already meet pre-established 
quality criteria, such as state preschool standards, Head Start standards, or national accreditation. However, 
there is criticism from stakeholders that alternative pathways reflect mistaken assumptions that all provid- 
ers within eligible groups have equivalent program quality. 

Regional Educational Laboratory Midwest Region states employed strategies to encourage early childhood education 
providers to join the quality rating and improvement system, such as linking state subsidies and other early childhood 
education funding to participation in the system 

REL Midwest Region states have encouraged providers to participate in QRISs and undertake quality 
improvements by offering a range of monetary incentives, including linking ratings to tiered reimburse- 
ment rates for child care subsidies or publicly funded centers, providing one-time cash awards to providers 
that receive a high rating or progress to a higher rating over time, or requiring a specific rating to receive 
other public funding such as state preK funding or local grants. Five of the seven REL Midwest Region 
states — Illinois, Indiana, Minnesota, Ohio, and Wisconsin — offer tiered reimbursements based on QRIS 
ratings to providers accepting at least some types of subsidies (table 5). 

Each state has an ongoing need to attract providers to participate in its QRIS and to encourage providers 
to continually improve their quality after an initial rating. Interviewees reported that incentive structures 
are most successful when they appeal to providers, foster participation in the system, and encourage pro- 
viders to invest in quality improvements. The use of tiered reimbursement rates to encourage enrollment is 
growing both in the REL Midwest Region and nationwide — for example: 

• Illinois offers higher tiered subsidy rates to providers that complete their QRIS submission by a 
particular date. 

• Indiana offers a time-limited opportunity for tiered reimbursement; only providers that apply to the 
QRIS by a certain date are eligible to receive ongoing tiered reimbursement rates. Subsidy reim- 
bursement rates for centers in the QRIS increase at level 1 (licensing) and at level 4 (accreditation). 

• Minnesota providers with three- or four-star ratings are eligible to receive tiered reimbursement 
up to 15 percent or 20 percent, respectively, above the standard maximum Child Care Assistance 
Program reimbursement rate. Only providers that participate in the state’s QRIS, Parent Aware, 
are eligible to offer scholarships to families. 3 

• Ohio offers tiered reimbursements for licensed child care centers that serve children who receive 
subsidies. 


Table 5. Most Regional Educational Laboratory Midwest Region states offer multiple incentives 
for providers participating in a quality rating and improvement system, and all states offer at least 
one type of incentive 


Incentive type 

Illinois 

Indiana 

Iowa 

Michigan 

Minnesota 

Ohio 

Wisconsin 

Tiered reimbursement of child care 

subsidies linked to participation and rating 

• 

• 



• 

• 

• 

One-time cash award for receiving a 
high rating or increasing rating level 


• 

• 



• 

• 

Eligibility or preference for certain grants 
or resources determined by rating 

• 


• 

• 

• 




Source: Authors' analysis based on a document review and transcripts from semi-structured interviews with quality rating and 
improvement system stakeholders (see appendix A). 
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• Wisconsin offers tiered subsidy payments based on the going market rate for early child care. Level 
1 providers are not eligible for subsidy reimbursements, Level 2 providers receive 95 percent of the 
market rate, level 3 providers receive the full market rate, level 4 providers receive 110 percent of 
the market rate, and level 5 providers receive 125 percent of the market rate. 

Most Regional Educational Laboratory Midwest Region states are actively revising their quality rating and 
improvement systems 

In all REL Midwest Region states QRIS development and refinement are ongoing, and systems are continu- 
ally evolving. All seven states have made at least one change to their system since inception — for example: 

• Illinois undertook a major redesign of its QRIS in 2013. Illinois initially developed Quality Counts 
in 2007, but after winning a Race to the Top-Early Learning Challenge grant from the U.S. 
Department of Education in 2013, Quality Counts gained momentum, underwent changes, and 
was renamed ExceleRate Illinois by the Illinois Network of Child Care Resource and Referral 
Agencies. 

• Iowa recalibrated its system in 2011 and is revising the standards again in 2015/16. 

• Michigan’s development of a QRIS began in 2005, was piloted in 2009, and was rolled out state- 
wide in 2012. The state revised the scoring criteria in June 2013. 

• Minnesota is considering changes to its system in 2015/16 to further refine Parent Aware. 

Implications and future considerations 


The study found that federal policies such as the Race to the Top-Early Learning Challenge grants influ- 
enced how and to what extent REL Midwest Region states developed or expanded QRISs. However, QRIS 
administrators still reported limited capacity and funding for full statewide implementation as a challenge 
even in states that received the grant. One key feature of any QRIS is how to measure quality. States across 
the REL Midwest Region used a combination of self-reported measures of structural quality and direct 
observations of process quality. The direct observations of process quality varied from published instru- 
ments, such as the Classroom Assessment Scoring System and the Environment Rating Scales — Revised, 
to locally developed measures used in Indiana and Ohio. These observations can be costly, and in an effort 
to reduce burden and cost, REL Midwest Region states either limited the use of direct observations to pro- 
grams eligible for higher ratings to specific types of programs or a subset of classrooms in larger centers or 
used alternate pathways to ratings. Alternate pathways to ratings, such as automatic or accelerated rating 
processes for programs with accreditation or public funding, were used in five out of seven REL Midwest 
Region states. These strategies allow states to retain their desired QRIS rating requirements, including 
measures of process quality, while reducing the cost and burden of the rating system. 

However, both strategies have the potential to change the QRIS rating that a program would have received 
otherwise. Reducing requirements for direct observations can affect the rating a program receives. For 
example, Karoly, Zellman, and Perlman (2013) found that ratings under the Colorado QRIS differed in 
approximately 30 percent of programs when only half of classrooms were randomly selected for observa- 
tion. And alternate pathways may lead to programs receiving a higher or lower rating than they otherwise 
would. For example, Kelton, Talan, and Bloom (2013) found that family child care providers under the 
Illinois QRIS were accelerated to the three-star rating because they were accredited and would not have 
earned it otherwise. 

REL Midwest Region states also used various types of financial incentives, such as tiered reimbursement 
for child care subsidies and bonus awards for ratings, to encourage programs to participate in the QRIS 
and increase their ratings over time. Given the expense of high-quality care, this strategy may encourage 
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providers to improve or maintain quality. However, states may want to exercise caution in using ratings 
to determine funding until there is evidence that the QRIS accurately differentiates low-, moderate-, and 
high-quality early child care. Validation studies, which link the rating with other data sources in the state, 
are necessary to document the extent to which the QRIS distinguishes provider quality and relates to chil- 
dren’s outcomes. Otherwise, states risk spending money on financial incentives that may fail to promote 
provider quality or children’s development. 

Finally, as Race to the Top-Early Learning Challenge grants expire, states may struggle to maintain 
forward momentum without federal funds to support their efforts, and issues of sustainability may arise. 
QRIS stakeholders may build support for sustainable funding in their own states by providing decision- 
makers with rigorous evidence on the validity of the QRIS ratings, as well as evidence to support financial 
incentives and cost-saving measures related to data collection. 

Additional resources to help states develop and 
refine their quality rating and improvement systems 


Findings from this report can help state QRIS administrators learn from each other’s successes and chal- 
lenges in implementing a QRIS. Additional resources to help states develop and refine their QRISs include: 

• The 2014 QRIS Online Compendium, A Catalog and Comparison of Quality Rating and Improve- 
ment Systems, which documents states’ attempts to promote thoughtful design, analysis, and 
ongoing improvement in early care and education systems (http://qriscompendium.org). 

• The website of the QRIS National Learning Network, a coalition of states and other organiza- 
tions formed to use rating and improvement strategies to improve children’s development (http:// 
qrisnetwork.org). 

• The REL Midwest Early Childhood Education Research Alliance website, which provides informa- 
tion on projects supporting the implementation of QRISs across the REL Midwest Region (http:// 
www.relmidwest.org/research-alliances/early'childhood'education-research'alliance). 

• Faria, Hawkinson, Greenberg, Howard, and Brown’s (2015) report on Michigan’s QRIS (http://eric. 
ed.gov/?id=ED555393). 
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Appendix A. Data collection and analysis methods 


This appendix describes the methods used to collect data for this study (document review and semi- 
structured telephone interviews) and to conduct the qualitative data analysis. 

Document review 

Before conducting the semi-structured interviews, the study team reviewed relevant documents on state 
QRIS websites, such as published reports, frequently asked questions, and guidance and administrative 
documents. The documents collected for the study included provider application templates, explanations of 
ratings structures, summaries that explained the QRIS to families, state survey response reports, and guid- 
ance for families and providers (table Al). The study team also used state profiles from the recently released 
QRIS compendium (QRIS Compendium, 2015). 

Semi-structured telephone interviews 

The data instrument was an interview protocol designed for use in a semi-structured interview format, with 
respondent review of prepopulated information for some protocol questions. The study team used the pro- 
tocol to collect detailed information on four QRIS aspects: the definition of quality, the method for calcu- 
lating provider ratings, quality improvement, and system features. The instrument contained 48 questions 
(some with subquestions) organized into six sections (background, quality, ratings, system, improvement 
plans, and conclusion). See appendix B for the full interview protocol. 

The semi-structured telephone interviews with QRIS representatives from all seven REL Midwest Region 
states took place between April and July 2013. Study team members experienced in semi-structured 


Table Al. Number and description of documents collected for each state 


State 

Number of documents 
collected 2 

Description of documents collected 

Illinois 

20 (2 from state agencies) 

Quality checklist, lists of participating providers, frequently asked questions web 
page, parent information web page, child care resource and referral information for 
providers, rating scales, overall list of number of providers participating in the QRIS, 
criteria for ratings, executive summary of the quality rating improvement system 
history and rating structure 

Indiana 

25 (4 from state) 

Levels of quality, system overview, system mission and goals, evaluation brief, 
standards, and validation study report 

Iowa 

17 (2 from state) 

Local child care resource and referral website, standard and criteria descriptions, 
Environment Rating Scales — Revised training materials, application forms, ratings 
summaries shared with providers 

Michigan 

19 (3 from state) 

Standards, presentation materials, Race to the Top-Early Learning Challenge application 

Minnesota 

16 (5 from state) 

Report on early learning in the state, validation study, presentation materials from 
Minnesota Professional Development Advisory Committee, Race to the Top-Early 
Learning Challenge application 

Ohio 

10 (2 from state) 

Validation study, Step Up to Quality guidance document (background and detailed 
information) 

Wisconsin 

29 (11 from state) 

Evaluation plan, provider measures, Department of Children and Families grant 
materials, social marketing campaign, point details, materials used to develop the 
quality rating and improvement system 

a. An example of one document may be a final report on the quality rating and improvement system methodology or a screenshot 
from one web page. 

Source: Authors' review of existing documents. 
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qualitative interviews conducted each 90-minute interview by telephone using the study’s interview pro- 
tocol instrument. A second study team member joined each interview to take notes. The interviews were 
also audio-recorded and transcribed by the study team. During the semi-structured interview the study 
team member followed the interview protocol questions in order, asked respondents to either answer ques- 
tions that had not been prepopulated or briefly verify the prepopulated responses, and updated information 
when necessary. If a question had already been answered during an earlier part of the interview, the ques- 
tion was read aloud and then skipped. After the interview, the study team reviewed the protocol responses 
and interview transcript to check for information that was not collected or confirmed during the telephone 
interview. The study team emailed questions that were missing responses to the respondent, with a request 
to provide the additional information as soon as possible. 

Qualitative data analysis 

Data analysis for the study used qualitative analysis methods and involved multiple steps, which included 
reviewing all documents to identify broad topic areas, developing a coding scheme to flag text pertain- 
ing to each topic, coding the data in qualitative analysis software to identify topics, generating reports to 
isolate each topic area, reading through all text coded in each topic report to find themes, and identifying 
findings for reporting in each theme area. In addition, some study data were more quantitative in nature 
(such as factual information about the types of providers eligible to participate in the QRIS or the types 
of quality improvement supports offered). The qualitative coding scheme focused on complex issues that 
are not easily quantified, and it did not include topics for some of the more objective information collected 
in the interview protocol. Instead, this objective information was reported in tables and state profiles and 
analyzed descriptively. 
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Appendix B. Semi-structured interview protocol 


This appendix includes the protocol used during the semi-structured interviews with state QRIS staff. The 
following is the script that was used as a guide. 

Final semi-structured interview protocol 

As you know, Regional Educational Laboratory (REL) Midwest is conducting a study to learn about how 
your state and other Midwestern states implement quality rating and improvement systems (QRISs). For 
this study, we are speaking with representatives from all of the Midwestern states serviced by REL Midwest; 
our aim is to understand how different states implement their QRISs. This information will help us to build 
a knowledge base about QRISs in the Midwestern states, which will be disseminated through a summary 
brief and a knowledge-sharing conference. We expect this information will be valuable to Midwestern 
states as they continue to roll out and refine their QRISs. 

Thank you for taking the time to review these questions and go through them with me over the phone at 
our scheduled time. We are asking you for detailed information about your state’s approach to developing 
and implementing a QRIS, including domains covered and your state’s measurement of quality, calculation 
of quality ratings, any quality improvement efforts in place in your state as part of your state’s QRIS, and 
how your state’s QRIS ratings are disseminated and used in your state. In some of the questions below, you 
will see answers based on what we found in our research. We have already conducted some preliminary 
research on your state’s QRIS and we would like to get clarification, correction, or confirmation from you 
about the information we have collected. 

We have a series of questions for you. Please answer to the best of your ability. 

Have you had the opportunity to read the materials that were sent to you via email? 

[Yes/no] 

Have you read the information page included in those materials and do you have any questions? 
[Interviewer waits for verbal acknowledgement] 

We would like your permission to record the interview so that we capture all the information verbatim. 
This recording will be shared only with the research team, stored in a secure server, and destroyed after the 
study. If there is anything you would like to have off the record, please let us know and we will not record 
it. Is it okay to move forward with recording the discussion? 

[Interviewer waits for verbal acknowledgement] 

Background information 

1. I’d like to confirm your role in your state. What is your title and what are your responsibilities? 

2. What is the name of your current quality rating and improvement system (QRIS)? 

[Confirm known name] 
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3. Is the QRIS a statewide system or is it implemented at a different level such as the county level? 

Probe: If not statewide, what is the geographic coverage of your QRIS? 

4- Are there any plans to change or update the QRIS? If so, what is the timing for those changes? 

5. We are interested in learning about the types of providers that are eligible to participate in the QRIS 
in [state]. I am going to read you a list of types of providers; please tell me if they are currently eligible 
to participate in the quality system. 

a. Are child care providers accepting child care subsidies eligible to participate (for example, vouch- 
ers)? [Yes/no] 

b. Are child care centers receiving direct subsidies from your state eligible to participate? [Yes/no] 

c. Are unsubsidized child care centers eligible to participate? [Yes/no] 

d. Are Head Start/Early Head Start programs eligible to participate? [Yes/no] 

e. Are state preschools eligible to participate? [Yes/no] 

f. Are licensed family child care homes eligible to participate? [Yes/no] 

g. Are legally nonlicensed providers eligible to participate? [Yes/no] 

h. Are programs funded by IDEA — Parts B and C eligible to participate? [Yes/no] 

i. Are Title I sites eligible to participate? [Yes/no] 

j. Are school-age programs eligible to participate? [Yes/no] 

k. Are other providers eligible to participate? [Yes/no]. Can you tell about these programs? 

6. Is the program voluntary for all programs in your state or are some types of programs required to partic- 
ipate? Are some programs exempt? Are some programs recruited? 

Probe for details [tailored based on document review] about which programs are required to participate or are 
exempt and under what circumstances. What mechanisms for recruiting are used? 

7. What are the minimum requirements for applying to be a part of the existing system? 

Probe: For example, must providers be licensed or is a minimum Environment Rating Scale (ERS) score 
required in order to receive a rating ? 

8. What age groups of children are included in the system? 

[If answered in questions 5j and 5k, skip to question 9.] 

9. Does participation occur at the program or classroom level? 
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Quality 


Thank you for that background information that provides helpful context. Now, the first topic we are going 
to talk about is Quality. 

10. What was the process that your state underwent to develop the QRIS? 

[Interviewer should try to keep this section short.] 

11. How does your state define “quality” for the QRIS? 

12. What aspects or dimensions of quality are measured in the QRIS? Does your state measure: 

a. Teacher qualifications (education and credentials)? [Yes/no] 

b. Curriculum? [Yes/no] 

c. Instructional quality (e.g., Classroom Assessment Scoring System [CLASS] scores)? [Yes/no] 

d. Classroom environment (e.g., Early Childhood Environment Rating Scale [ECERS])? [Yes/no] 

e. Teacher warmth and sensitivity? [Yes/no] 

f. Staff-child ratio and group size? [Yes/no] 

g. Accreditation? [Yes/no] 

h. Parent involvement/family engagement? [Yes/no] 

i. Community partnerships (e.g., fit of the program model with the community’s family needs)? [Yes/no] 

j. Program leadership/management/teacher professional development supports? [Yes/no] 

k. Cultural and linguistic diversity? [Yes/no] 

l. Child assessment/developmental screening? [Yes/no] 

m. Provisions for children with special needs? [Yes/no] 

n. Transition to kindergarten? [Yes/no] 

o. Licensing compliance? [Yes/no] 

p. Other? [Yes/no] Can you tell us about these? 

13. Please tell me about the process of selecting aspects or dimensions of quality to include in the QRIS. 
How were these dimensions selected? 
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a. Who was involved in defining quality? Did your state work with an advisory panel or consultant(s)? 
Describe those involved (e.g., state policymakers, academics, practitioners, parent representatives). 

b. What sources of information or literature did you draw on to select these dimensions? 

c. Did you choose any aspects of quality to measure based on widely used instruments such as ECERS 
or CLASS? 

Probe: Were any suggestions discarded ? 

d. Do you have documentation that describes the process for selecting dimensions of quality? 

14- Does the QRIS include classroom observations to measure quality? 

[If yes, continue to questions 14a— 14d. If no, skip to Ratings section] 

a. What instruments are used (ECERS, Family Day Care Rating System [FDCRS], Infant/Toddler 
Environment Rating Scale [ITERS], CLASS, Arnett Caregiver Interaction Scale [CIS], other)? 

b. Are the observations. . . 

i. Self-reported by the program? [Yes/no]. 

ii. Conducted by independent observers? [Yes/no]. 

c. Are observers trained to have inter-rater reliability? What kind of training do they receive? 

d. How did your state make the decision to include self-reported or independent observations of 
quality in the QRIS? 

Ratings 

Next, we would like to learn more about the Ratings used in your QRIS. 

15. Based on the documentation we reviewed about your state’s QRIS, we see that your state calculates 
the ratings as such [discuss/explain the calculation of state’s rating from review of documentation]. Is this 
correct? 

16. Are ratings calculated using a block system, point system, or a combination? 

17. Does your rating system vary by program or provider type? If so, how? 

Probe: For example, are licensed family child care homes rated on a different set of dimensions or with a differ - 
ent calculation approach than other licensed centers 1 

a. Do all providers participating in the QRIS receive a rating? [Yes/no]. 
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18. We just discussed dimensions or aspects of quality in question 12. Did you set up your ratings to priori- 

tize certain aspects of quality that are more important? 

Probe: In other words, are some dimensions weighted more than others in the rating calculation 1 If so, which 

ones and how is the weighting done 1 

19. Please describe the process of determining the rating calculation approach. 

a. Who were the stakeholders at the table during this process and who had primary authority to make 
decisions? Did your state work with an advisory panel, organization, or consultant(s)? Describe those 
involved in the process (e.g., state policymakers, academics, practitioners, parent representatives). 

b. How easy or difficult was it to arrive at a consensus for the rating system? Were there any topics 
that sparked disagreements or intense discussions? 

c. How did your state determine the cutoff scores for each rating level? What sources of information 
were used to select the cutoff scores and how much of this decision was based on empirical research 
evidence versus recommendations from experts? 

20. Is there a process through which providers can appeal their rating? 

21. How long is the program rating valid before it must be reassessed? 

22. Did you pilot the rating system? 

Probe: Can you tell me about how you piloted the scoring system ? 

23. Were the ratings validated using the pilot or other data collection? 

24- Do providers receive technical assistance to achieve the minimal ratings required or desired for partici- 
pation, or for readiness for participation? 

System 

25. Are QRIS ratings made available to the general public for ALL participating programs? 

[If yes, skip to question 27. If no, continue to questions 25a— 25c] 

a. If not, which program ratings are publicly available and which are not? 

b. For ratings not available to the general public, who has access to the ratings data and under what 
circumstances? 

c. If ratings are not publicly available, can parents have access to the ratings under specific circum- 
stances? If so, please describe. 

[If any ratings are made available to the public or to parents at all, go to question 26. Otherwise, skip to 
question 29.] 
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26. What efforts has your state made to ensure that families are aware that this resource exists? Please tell 
me about the dissemination plan. 

27. How do you disseminate the QRIS data to parents or other stakeholders? 

a. Do you . . . 

i. Disseminate information through child care resource and referral agencies? [Yes/no] 

ii. Disseminate information through a dedicated state QRIS website? [Yes/no] 

iii. Disseminate information through other websites? [Yes/no] 

iv. Disseminate information through social service agencies or caseworkers? [Yes/no] 

v. Disseminate information through other source(s)? [Yes/no] Please explain. 

b. If families do not have Internet access, what other ways are they are able to access ratings 
information? 

28. What level of information about program quality is available to consumers? 

Probe: For example, in addition to the overall program quality rating level, do parents have access to provider 
scores on the individual quality measures or to subratings in specific quality domains ? 

29. What data are collected on providers in the system and on families using those providers? 

Probe: Does your state collect and store data about program characteristics (such as program type and model, 
number of children and ages served, service area and location, funding source) in addition to the QRIS ratings 1 

Probe: Does your state collect and store data about children and families attending programs that participate 
in the QRIS? 

Probe: How are the QRIS data linked with other longitudinal data systems? Are children or staff tracked 
using unique IDs, and can those IDs be linked to other state data systems such as public schools? 

30. We are interested in finding out how the QRIS ratings are utilized by your state. Does your state use 
ratings to: 

[Read each line to the respondent] 

a. Determine provider eligibility to accept child care subsidy vouchers? [Yes/no] 

b. Determine the reimbursement rate for providers accepting subsidies? [Yes/no] 

c. Qualify providers for state preschool or other direct payment contracts with your state? [Yes/no] 

d. Determine funding levels for providers participating in state preschools or other state contracts? 
[Yes/no] 
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e. Prioritize highly rated programs for other public or private grants? [Yes/no] 

f. Determine professional development needs statewide? [Yes/no] 

g. Determine professional development needs for individual programs? At the classroom level? [Yes/no] 

h. Determine other material-, intervention-, or curriculum-based needs? [Yes/no] 

i. Other? [Yes/no] Please explain. 

31. Are the ratings high stakes for any programs (e.g., does funding for some program types depend on the 
quality rating)? 

Improvement plans 

Next, we would like to learn more about the Improvement Plans or other quality improvement efforts that 
are part of the QRIS. 

32. Does your state include quality improvement plans (or other quality improvement supports) as part of 
the QRIS? 

[If no, skip to Conclusion; we will ask about other quality initiatives at the end], 

33. Are improvement plans (or other QRIS quality improvement supports) available to all participating 
providers or only to certain ones? 

a. If only certain providers, please explain which ones (e.g., subsidized providers, state preK). 

b. Can unlicensed providers or licensed family child care providers participate? 

c. Is there support for those who don’t qualify (preliminary support to get up to speed)? 

34- Are improvement plans (or other QRIS quality improvement supports) triggered by a particular quality 
rating level or can eligible providers participate in improvement plans regardless of the rating level? If 
so, please describe. 

Probe: Are certain types of programs, such as state preK, required to participate in an improvement plan 
under certain circumstances? 

35. Are there any other criteria used to determine eligibility for participation in improvement plans (or 
other QRIS quality improvement supports)? 

36. Please describe the process within the QRIS for developing an improvement plan (or planning for 
other QRIS quality improvement supports). 

a. How are the specific aspects or domains of quality that need improvement identified? Are the 
results of the QRIS rating measures used or is there an additional assessment process? If so, who 
completes the assessment — the provider or an external assessor/consultant? 
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b. Which stakeholders are involved in the improvement plan process (e.g., teachers, parents, commu- 
nity members)? 

c. Do improvement plans entail a written document summarizing planned quality improvement 
activities? If so, who completes the written plan — the provider or an external assessor/consultant? 

37. Can you tell me about the content of the improvement plans (or other QRIS quality supports/ improve- 
ment supports)? 

Probe: Are certain aspects or domains of quality prioritized over others for improvement plans (e.g., provider 
qualifications, instructional quality, family involvement )? 

38. What are the strategies and methods for delivery of the improvement plans? Do your improvement 
plans involve: 

a. Technical assistance/consulting support [Yes/no] 

b. Training/workshops [Yes/no] 

39. Who provides and pays for the improvement assistance? 

40. Is compliance with or successful completion of the improvement plan (or other quality improvement 
supports) mandatory or high stakes (e.g., dictates funding) for any providers? If so, which providers and 
what are the stakes? 

Conclusion 

41. Have any validation studies been conducted by your state or by outside researchers of your state’s QRIS? 
If yes, could you please share them with us? 

Probe: Ask about pilot studies if they did not mention them above in quality section. 

42. How did you build political will and buy-in for the QRIS in your state? Please briefly list these activities. 

43. Can you please share with us any challenges you faced in developing and implementing the QRIS in 
your state (beyond those you shared in the previous question)? 

44. Or helpful tips you discovered? 

45. Is your state engaging in any quality improvement activities that you or others believe are exceptionally 
effective at improving program quality or serving as a model for other states? If yes, what are they? 

46. Aside from eligibility for publicly funded programs or tiered reimbursement rates, does your state offer 
any other financial or nonfinancial incentives to improve program quality? For example, does your state 
offer: 

a. Start-up grants? [Yes/no] 

b. Quality improvement awards? [Yes/no] 


21 


c. Scholarships for children? [Yes/no] 

d. Curriculum/instructional improvement grants? [Yes/no] 

e. Facility improvement grants? [Yes/no] 

f. Staff scholarships/tuition assistance? [Yes/no] 

g. Staff wage enhancement/retention bonus/stipend? [Yes/no] 

h. Teacher permit reimbursement? [Yes/no] 

i. Publicizing of QRIS ratings? [Yes/no] 

j. Other? [Yes/no] Please describe. 

47. Are there any additional documents and/or sample reports that come to mind now that you have not 
yet shared with us that would help us gain a better understanding of the QRIS used in your state? 

48. Is it okay if we follow up with you by telephone or email if we need clarification on anything? 

If there is anything about your state’s QRIS that you would like to add that we haven’t covered during this 
conversation, please feel free to contact us any time. Thank you! 
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Notes 


1. See http://www2.ed.gov/programs/racetothetop'earlylearningchallenge/index.html and https://www. 
whitehouse.gov/the'presS'office/2013/02/13/fact'sheet'president'obama'S'plan'early'education'all 
-americans for more information on these initiatives. 

2. Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, and Wisconsin. 

3. The Minnesota Department of Education offers Early Learning Scholarships that can be used to cover 
family co-payments, rates, or fees. Elowever, the funds can be applied only to care or education provid- 
ed by programs that participate in Parent Aware. 
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